Papers with compromised reasoning
Self-contradictory reasoning evaluation and detection (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown impressive reasoning ability, but many downstream reasoning tasks focus on performance-wise evaluation. |
| Approach: | They define and assess the Self-Contra rate across three datasets and delve into finer-grained categories of Self-contra reasoning. |
| Outcome: | The proposed model can detect self-contra reasoning with a 52.2% F1 score, much lower than for humans. |